System and method for identifying target objects

ABSTRACT

A method, system, article of manufacture, and UAV configured to identify a target object shown in an image, such as shown in a perspective view that is a two-dimensional image or frame of video. The method comprises: identifying and track the position of a target object shown in a sequence of images or video, even when the target object may be traveling at high speeds, detecting the target object within an image based on one or more of the object&#39;s physical characteristics, such as its color, shape, size, chrominance, luminance, brightness, lightness, darkness, and/or other characteristics. Thus, in this context a target object may be anything having one or more detectable physical characteristics. The method also providing an improved and more intuitive user interface that enables a user to select a target object for tracking. As a result, the method and system improve the accuracy, usability, and robustness of the system.

TECHNICAL FIELD

The present disclosure relates generally to visual tracking of targetobjects and, more particularly, to systems and methods of identifyingtarget objects.

BACKGROUND

Unmanned aerial vehicles (“UAV”), sometimes referred to as “drones,”include pilotless aircraft of various sizes and configurations that canbe remotely operated by a user or programmed for automated flight. UAVscan be used for many purposes and are often used in a wide variety ofpersonal, commercial, and tactical applications. For instance, UAVs canbe equipped with imaging equipment, such as cameras, video cameras,etc., which allow users to capture images or video footage that is toodifficult, not practical, or simply not possible to capture otherwise.UAVs equipped with imaging devices find particular use in thesurveillance, national defense, and professional videography industries,among others, besides being popular with hobbyists and for recreationalpurposes.

UAVs equipped with imaging equipment may allow users to track a targetobject remotely. This ability to track a target object allows the UAVsto operate autonomously while tracking the movement of the object.However, tracking a target object using UAVs may present challenges. Forexample, there is an existing need for UAVs that can accurately track atarget object travelling at high speed relative to other objects andbackground scenery, as well as target objects with various shapes.

SUMMARY

The disclosed embodiments include methods, systems, articles ofmanufacture, and UAVs configured to identify a target object shown in animage, such as shown in a perspective view that is a two-dimensionalimage or frame of video. The techniques described in the disclosedembodiments may be used to identify and track the position of a targetobject shown in a sequence of images or video, even when the targetobject may be travelling at high speeds. The disclosed embodimentsdetect the target object within an image based on one or more of theobject's physical characteristics, such as its color, shape, size,chrominance, luminance, brightness, lightness, darkness, and/or othercharacteristics. Thus, in this context a target object may be anythinghaving one or more detectable physical characteristics. The disclosedembodiments also provide an improved and more intuitive user interfacethat enables a user to select a target object for tracking. As a result,the disclosed embodiments improve the accuracy, usability, androbustness of the system.

In the disclosed embodiments, a system may receive a user inputindicating the position of a target object within an image, for example,based on a user-selected point, pixel, region, area, or coordinates inthe image. The system may define a first area and a second area in theimage based on the user-selected position. The system may compare imagecharacteristics in the first and second areas to identify the targetobject within the image. The image may be a two-dimensional perspectiveview of image or video data captured by a movable device, such as a UAV.

In one aspect, the disclosed embodiments may define the first area as aforeground area and the second area as a background area surrounding theforeground area. In one aspect, the disclosed embodiments may determinea representation of a first physical meaning of objects and featuresshown in the image based on image characteristics in the first area anda representation of a second physical meaning based on the imagecharacteristics in the second area. The disclosed embodiments maycompare the representations of the first and second physical meanings toidentify the target object in the image.

In another aspect, the disclosed embodiments may generate a firsthistogram representing the image characteristics in the first area and asecond histogram representing the image characteristics in the secondarea, determine a third histogram by combining the first and secondhistograms using a predetermined function, and apply the third histogramto an area of interest, the area of interest comprising the first andsecond areas. In one aspect, the predetermined function may determine,for each image characteristic, a ratio of a value for that imagecharacteristic in the first area divided by a value for that imagecharacteristic in both the first and second areas.

In a further aspect, the disclosed embodiments may redefine the firstand second areas based on the probabilities of whether points, pixels,regions, or areas in the first and second areas contain the targetobject. In a further aspect, the disclosed embodiments may generate anew first histogram representing the image characteristics in theredefined first area and a new second histogram representing the imagecharacteristics in the redefined second area, determine a new thirdhistogram by combining the new first and second histograms using asecond predetermined function. In one aspect, the disclosed embodimentsmay use the same predetermined function to determine the third histogramand the new third histogram. In a further aspect, the disclosedembodiments may apply the new third histogram to the area of interest tocreate a back-projected image.

In a further aspect, the disclosed embodiments may repeat each of thesteps of redefining the first and second areas, generating a new firsthistogram and new second histogram, determining a new third histogram,and applying the new third histogram to the area of interest. In oneaspect, the disclosed embodiments may identify likely contours of thetarget object. In one aspect, the disclosed embodiments may use theidentified target object to track a position of the target object.

The techniques described in the disclosed embodiments may be perforatedby any apparatus, system, or article of manufacture, including a movableobject such as a UAV, or a controller, or any other system configured toreceive image data (including video data) and track target objects shownin the received images. Unlike prior tracking systems, the techniquesdescribed herein can more accurately track target objects that may bemoving at high speeds relative to the image-capture device.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the disclosed embodiments as defined in theclaims.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate several embodiments and, togetherwith the description, serve to explain the disclosed principles. In thedrawings:

FIG. 1 is a schematic diagram of an exemplary system that may be used toidentify a target object in accordance with the disclosed embodiments.

FIG. 2 is a schematic diagram of another exemplary system that may beused to identify a target object in accordance with the disclosedembodiments.

FIG. 3 is a schematic diagram of yet another exemplary system that maybe used to identify a target object in accordance with the disclosedembodiments.

FIG. 4 is a flowchart illustrating an exemplary sequence of steps thatmay be performed for identifying a target object in accordance with thedisclosed embodiments.

FIG. 5 is a schematic diagram of an exemplary perspective view havingfirst and second areas in accordance with the disclosed embodiments.

FIG. 6 is a schematic diagram of another exemplary perspective viewhaving first and second areas in accordance with the disclosedembodiments.

FIG. 7 is an exemplary perspective view that may be used to identify atarget object in accordance with the disclosed embodiments.

FIGS. 8a-d are exemplary views showing various stages of imageprocessing for identifying a target object within the exemplaryperspective view of FIG. 7 in accordance with the disclosed embodiments.

DETAILED DESCRIPTION

Tracking a target object using UAVs may present challenges. For example,first, the target object may be travelling at a high speed, so the UAVmust be able to identify the target object quickly without losingaccuracy in its tracking; second, when a user identifies a target objectfor the UAV to track, the user often cannot accurately select the targetobject if its contour is not in a regular shape (such as a square,rectangular, etc.), which actually accounts for most cases in practicaluse. The known technologies simply instruct the UAVs to track whateverclosed area the user selects, without approximating to the contours ofthe target object in the image. As a result, objects other than thetarget object are being tracked by the UAV, including features inbackground scenery or the surrounding area. This renders the trackingless responsive and prone to losing sight of the target object.

The disclosed embodiments provide improved techniques for visualtracking of target objects and, more particularly, systems and methodsof identifying target objects in perspective views based on a userselection. The resulting systems and methods provide enhanced accuracy,usability, and robustness in their ability to identify a target object,which may be moving at a high speed, in a perspective view.

Reference will now be made in detail to exemplary disclosed embodiments,examples of which are illustrated in the accompanying drawings anddisclosed herein. Where convenient, the same reference numbers will beused throughout the drawings to refer to the same or like parts.

FIG. 1 is a diagram of an exemplary system 100 for performing one ormore operations in accordance with the disclosed embodiments. The system100 may include one or more movable objects 102 of various sizes andconfigurations. According to some embodiments, the movable object 102may be any suitable object, device, mechanism, system, or machineconfigured to travel on or within a suitable medium (e.g., a surface,air, water, rails, space, underground, etc.). The movable object 102 maybe an unmanned aerial vehicle (UAV), for example, that is movable usingone or more propellers 110. Although the movable object 102 is shown anddescribed herein as a UAV for exemplary purposes of this description, itwill be understood that other types of movable objects (e.g., wheeledobjects, nautical objects, locomotive objects, other aerial objects, orthe like) may also or alternatively be used in embodiments consistentwith this disclosure. As used herein, the term UAV may refer to anaerial device configured to be operated and controlled autonomously(i.e., via an electronic control system) and/or manually by off-boardpersonnel.

The movable object 102 may be configured with imaging equipment 106,such as cameras, video cameras, or the like, to capture and track atarget object. In some embodiments, the movable object 102 may includeone or more processors, one or more input/output (I/O) devices, and oneor more memories. The movable object 102 may also be configured tocommunicate with one or more other components in the system 100including, for example, a user controller 104 or other movable objects(not shown).

As shown in FIG. 1, the movable object 102 may communicate with the usercontroller 104, for example, over a wireless link. The movable object102 may include an interface 108 for communicating with the usercontroller 104 via any appropriate wireless protocols. The usercontroller 104 may include, but is not limited to, a general-purposecomputer, computer cluster, terminal, mainframe, mobile computingdevice, or other computer device capable of receiving user input. Inthis context, a mobile computing device may include, but is not limitedto, a mobile phone, smartphone, personal digital assistant, tablet,laptop, etc.

The user controller 104 may be configured to communicate with one ormore components of the system 100 including, for example, the movableobject 102, other UAVs (not shown), and other user controllers (notshown). In some embodiments, the user controller 104 may executesoftware configured to communicate with the movable object 102, displayone or more images obtained from the imaging equipment 106 on themovable object 102, and receive user inputs (e.g., to select targetobjects in displayed images or video) through an input device, such as akeyboard, touchscreen, mouse, stylus, or any other device or combinationof devices through which the user can provide input data. In someembodiments, the disclosed operations for identifying a target objectare performed by software in the movable object 102, but alternatively,these operations may be performed by software in the user controller104, or may be performed by the coordinated operation of softwareexecuting in the movable object 102 and software executing in the usercontroller 104.

FIG. 2 is a schematic block diagram of an exemplary system 200 that maybe used in accordance with the disclosed embodiments. The system 200, orvariations thereof, may be used to implement components in the system100, including for example the movable object 102 and/or user controller104. The system 200 may include one or more processors 220, one or moreI/O devices 222, and one or more memories 224, which in some embodimentsmay be implemented within one or more controllers 210. In someembodiments, the system 200 may take the form of a mobile computingdevice, general-purpose computer, a mainframe computer, or anycombination of these components. Alternatively, the system 200 may beimplemented in a UAV 102 or a user controller 104 for performing one ormore operations consistent with the disclosed embodiments. For example,the system 200 may be a stand-alone system, or it may be implemented asa subsystem in a larger system, where one or more operations in thesystem 200 are performed using parts of the larger system.

Processor 220 may include one or more known processing devices. Forexample, the processor may be from the family of processors manufacturedby Intel, from the family of processors manufactured by Advanced MicroDevices, or the like. Alternatively, the processor may be based on theARM architecture. In some embodiments, the processor may be a mobileprocessor. The disclosed embodiments are not limited to any type ofprocessor configured in controller 210.

I/O devices 222 may be one or more devices configured to allow data tobe received and/or transmitted by the controller 210. The I/O devices222 may include one or more communication devices and interfaces, andany necessary analog-to-digital and digital-to-analog converters, tocommunicate with other machines and devices, such as other components inthe system 100, including the movable object 102 and/or user controller104.

Memory 224 may include one or more storage devices configured to storesoftware instructions used by the processor 220 to perform functionsrelated to the disclosed embodiments. For example, the memory 224 may beconfigured to store software instructions, such as program(s) 226, thatperform one or more operations when executed by the processor(s) 220 toidentify a target object in an image. The disclosed embodiments are notlimited to software programs or devices configured to perform dedicatedtasks. For example, the memory 224 may include a single program 226,such as a user-level application, that performs the functions of thedisclosed embodiments, or may comprise multiple software programs.Additionally, the processor 220 may execute one or more programs (orportions thereof) remotely located from the controller 210. For example,the movable object 102 may access one or more remote softwareapplications via the user controller 104, such that, when executed, theremote applications perform at least some of the functions related tothe disclosed embodiments for identifying a target object. Furthermore,the memory 224 also may be configured to store data, for example, foruse by the software program(s) 226.

It is to be understood that the configurations and boundaries of thefunctional building blocks shown for exemplary systems 100 and 200 havebeen arbitrarily defined herein for the convenience of the description.Alternative implementations may be defined so long as the specifiedfunctions and relationships thereof are appropriately performed and fallwithin the scope and spirit of the invention.

FIG. 3 is a diagram of another exemplary system 300 for identifying atarget object in accordance with disclosed embodiments. As shown in FIG.3, a user controller 304 may include a display device for displayinginfoo tation to a user and may also include one or more input devicesthrough which the user can input data and/or control operation of aremote UAV 302. The user controller 304 may be configured to receivesignals from the UAV 302, where the received signals may be indicativeof information or data relating to movements of the UAV 302 and/or data(e.g., imaging data) captured using imaging equipment on the UAV 302. Insome embodiments, the user controller's display device may be amultifunctional display device, such as a touch screen, that isconfigured to display information and also receive user inputs. Forexample, in one embodiment, the display device may be configured toreceive user inputs via a multifunctional screen. In another embodiment,the multifunctional screen may be the only input device for receivinguser input.

In some disclosed embodiments, the display device may be an integralcomponent of the user controller 304. That is, the display device may bebuilt-in, attached, or fixed to the user controller 304. In otherembodiments, the display device may be connectable to (anddis-connectable from) the user controller 304. For example, the usercontroller 304 may be configured to be electronically connectable to adisplay device (e.g., via a connection port or a wireless communicationlink), or the display device may be connectable to the user controller304 via a mounting device, such as by a clamping, clipping, clasping,hooking, adhering, or other type of mounting device.

As shown in FIG. 3, a user may see a stream of image or video data thatwas captured by the UAV 302, transmitted to the user controller 304, andpresented to the user as a two-dimensional perspective view on thedisplay device. In some embodiments, the stream of images or video isdisplayed on a built-in display device in the user controller 304, oralternatively the stream may be displayed on an external display deviceconnected to, or in communication with, the user controller 304, such ason a display of a smartphone, tablet, laptop, monitor, television, orany other display device that may be coupled to the user controller 304.In other embodiments, the image or video stream may be accessed throughan application executing on a smartphone, tablet, or the like.

FIG. 4 shows a flowchart illustrating a sequence of steps that performsan exemplary process 400 for identifying a target object in accordancewith the disclosed embodiments. The process of FIG. 4 may be implementedin software, hardware, or any combination thereof. For purposes ofexplanation and not limitation, the process 400 will be described in thecontext of system 100, such that the disclosed process may be perforatedby software executing in the user controller 104 and/or movable object102.

In the system 100, the user controller 104 may receive a live ortime-delayed stream of image or video data from the movable object 102.The user controller 104 may display the received image or video data ina perspective view on a display device, for example, built into the usercontroller. The perspective view may correspond to any image obtainedfrom a camera or video equipment, for example, on the movable object102. According to some embodiments, the user operating the usercontroller 104 may see a target object that the user desires to track inat least one of the displayed perspective views of image or video data.The target object, for example, may include stationary objects such asparked cars, buildings, geographic features, etc. Alternatively, thetarget object may be slow-moving objects such as a person on a hike, ora fast-moving object such as a moving vehicle.

Having identified a target object in a displayed perspective view ofimage or video data, the user may select the target object in the viewusing an appropriate input device, for example, connected to orintegrated in the user controller 104. At step 410, the system mayreceive the user input, including the user selection information. Theuser section information may be presented in various styles or forms. Inan exemplary embodiment, the user may select a point on the targetobject (e.g., point 5 a on a target object in FIG. 5) using, forexample, a touchscreen or mouse configured to select a point in thedisplayed data. In another embodiment, the user may drag and select anarea containing the target object (e.g., area 6 b including a targetobject in FIG. 6) formed, for example, by dragging a cursor along path 6a or by the user drawing a rectangular perimeter 6 b around the targetobject. For example, when the target object is a fast-moving vehicle,the user may select the vehicle as a target object by simply clicking ona point on the displayed vehicle using a mouse or similar input devicerather than selecting the entire vehicle. However, when the targetobject is stationary or moving slowly, the user alternatively may selectthe entire target object by selecting an area around the object.

After receiving the user input at step 410, at step 420 the system mayperfoini an initialization process in which it may define an area ofinterest (e.g., the area within perimeter 5 b in FIG. 5) based on theuser-selected point (e.g., point 5 a in FIG. 5). In defining the area ofinterest, the system 100 may consider various factors such as the totalresolution of the video feed, the system's maximum target sizerequirement, the minimum details needed for accurate identification,etc.

At step 420, the system 100 may determine a foreground area based on thearea of interest (e.g., foreground area is the area within perimeter 5 cin FIG. 5). In defining the foreground area, the system may considervarious factors such as the total resolution of the video feed, thesystem's maximum target size requirement, the minimum details needed foraccurate identification, etc. The size and shape of the foreground areamay be arbitrarily defined, and the system 100 may define the foregroundarea using various sizes or shapes in the displayed perspective view,and in some embodiments the foreground area may comprise a collection ofdifferent areas that collectively form the foreground area. In someexemplary embodiments, the width and height of the foreground area maybe chosen to be 50% of the width and height of the area of interest. Inother embodiments, the foreground area has the user-selected point atits center. During this initialization process, the system 100 also maydefine a background area relative to the foreground area and the area ofinterest (e.g., the background area in FIG. 5 is the area between thebounding perimeters 5 b and 5 c).

In other embodiments, the system 100 may define the area of interestbased on a user-selected area, for example, selected by a user who dragsa cursor to select an area that encompasses the desired target object inthe perspective view. For example, the system 100 may set theuser-selected area as the outer boundary of the area of interest (e.g.,perimeter 6 b in FIG. 6). System 100 may define the foreground areawithin the area of interest (e.g., the area within the perimeter 6 c inFIG. 6) and a corresponding background area (e.g., the area between theperimeters 6 b and 6 c).

When the area of interest is defined based on a user-selected area, itis possible to assume that the probability of the target object beingnear the center of the area of interest is greater than the probabilityof the target object being in the background area, which would be closerto the outer boundary of the area of interest. In some embodiments, thesystem 100 may define the foreground area as an area covering the centerof the area of interest and also define the background area as an areanear the outer boundary of the area of interest. Additional detailsrelated to exemplary step 420 are illustrated in FIGS. 5 and 6.

Next, at step 430, the system 100 may determine histograms F(hist) andB(hist) corresponding to the displayed image or video data within eachof the foreground and background areas that were defined at step 420. Inessence, and as explained further below, the system 100 attempts to mapthe physical meanings of the image characteristics found in theforeground and background areas in respective histogram representations.These histograms represent the distributions of pixels having certainimage characteristics contained in the foreground and background areasand their respective physical meanings.

In some embodiments, the system 100 may determine the most suitablehistogram format based on the existing lighting condition and/oravailable color range. For example, in a low-lighting condition wherethe color range is limited, an intensity histogram may be adequate tocharacterize the image's pixel values in each of the foreground andbackground areas. Alternatively, an RGB color space, for example, may beused to characterize the image characteristics and generate histogramsfor the foreground and background areas. In other embodiments, thesystem 100 may determine the histograms based on other color spacesand/or image characteristics. For example, the histogram may representthe distribution of an area's hue, saturation, and value in an HSVspace. Accordingly, other image characteristics in the foreground andbackground areas also may be described by a histogram, including but notlimited to, the chrominance, luminance, brightness, lightness, darkness,etc.

In an exemplary embodiment, the system 100 may separately create ahistogram for each of the foreground and background areas.Alternatively, the system 100 may simultaneously create the histogramsfor the foreground and background areas. For purposes of thisdisclosure, the histogram of the foreground area may be represented by afunction F(hist), and the histogram of the background area by a functionB(hist). In some embodiments, the system 100 may create separatehistograms of red, green, and blue channels (in a RGB color space) foreach of the foreground and background histograms, or alternatively thesystem 100 may generate a three-dimensional (3D) RGB histogram with itsthree axes representing the red, green, and blue channels, for each ofthe foreground and background areas. For example, F(hist) may be a 3DRGB histogram of the foreground area, where each pixel value in theforeground area is represented by a red, green, and blue (RGB)coordinate in the 3D space. In such an embodiment, the histogram F(hist)would reflect the number of pixels in the foreground area for eachpossible RGB coordinate. Similarly, the histogram B(hist) may be a 3DRGB histogram for the pixel values in the background area, where eachpixel value of the background area is represented by RGB coordinate. Insome embodiments, the system 100 may normalize the F(hist) and B(hist)histograms, depending on the sizes of the foreground and backgroundareas, so histogram F(hist) and histogram B(hist) may be of similarsizes and/or shapes.

As an example, the foreground area may primarily contain the targetobject (e.g., the car inside the foreground area bounded by perimeter 5b in FIG. 5). If the target object is a red sports car, for example,then the 3D RGB histogram F(hist) will reflect that with a relativelyhigh count of red pixels. That is, the majority of pixels in theforeground area will be distributed on the red axis of the histogramF(hist). Similarly, if the target object is blue, then histogram F(hist)will reflect a majority of pixels along the blue axis. Thus, histogramF(hist) not only represents the image characteristics of the foregroundarea but may also represent the one or more physical characteristics ofthe target object, such as the object's color in this example.

Similarly, histogram B(hist) represents the image characteristics of thebackground area. Because the probability of the target object existingin the background area may be relatively low, histogram B(hist) mayrepresent other non-target objects and features, such as backgroundscenery. In the example of a moving vehicle, the background area maycomprise features such as trees, roads, buildings, people, etc., thatare not target objects being tracked. Accordingly, the distribution ofpixels in B(hist) could be spread evenly across the 3D space (meaningthere is an even distribution of colors in the background), or primarilyin the gray color zone (e.g., representing the color of road pavement inthe background area), or may comprise other possible distributionsdepending on what objects and features are in the background area.

After obtaining the histograms F(hist) and B(hist), at step 440 thesystem 100 may compare the image characteristics of these foreground andbackground histograms to identify the target object, such as thecontours of the target object. In some embodiments, the system 100 maycompare the image characteristics by calculating a new histogramNF(hist) according to the formula below:

$\begin{matrix}{{{NF}({hist})} = \frac{F({hist})}{{F({hist})} + {B({hist})}}} & (1)\end{matrix}$

The histogram NF(hist) in equation (1) above represents one possiblecomparison between the histograms F(hist) and B(hist), where each valuein the histogram NF(hist) has a value between zero and one. In someembodiments, after the system 100 creates the histogram NF(hist), it mayfurther normalize the values in NF(hist) to a range between zero to oneif they were not already in this range. In other embodiments, the system100 may scale or normalize the values in NF(hist) to fit within otherdesired value ranges. Alternatively, the system 100 may utilize othersaliency detection formulas. In general, the function used for NF(hist)may be selected based on the type of histogram (e.g., 3D RGB space, HSVspace, chrominance, luminance, brightness, lightness, darkness, etc.)and the physical characteristics used to track the target object in theperspective view.

Several assumptions may be made about NF(hist) based on formula (1). Forexample, if the foreground area contains the target object but thebackground area does not, then NF(hist) may comprise a relatively largevalue, for example greater than 0.5, in the portion of the histogramcorresponding to a physical characteristic of the target object. Theopposite assumption applies if the background area contains the objectbut the foreground area does not, whereby NF(hist) will consist ofrelatively small values, for example less than 0.5. Further, if theforeground area and the background area both contain the target object,then the values of NF(hist) may fall within the middle of the range,e.g., around 0.5. In a disclosed embodiment using a 3D RGB color spacefor F(hist) and B(hist), it is possible to determine the probabilitywhether or not a particular (R, G, B) element in the histogram NF(hist)represents the target object or not, as discussed further below.

At step 450, the system 100 back-projects the histogram NF(hist) ontothe area of interest (e.g., onto the area of interest bounded by theperimeter 5 b in FIG. 5). In some embodiments, the back-projectionprocess may comprise analyzing each pixel value in the area of interestand reassigning the pixel's image characteristics based on itscorresponding value in the histogram NF(hist), resulting in a new imagein which every pixel in the area of interest is scaled between zero andone, based on its corresponding value in the histogram NF(hist). In someembodiments, the system 100 previously normalized the values inhistogram NF(hist), so the pixel values of the new image are also in thenormalized range. For example, as a result of the back-projectionprocess in step 450, the new image may appear as a gray-scaled versionof the original image, where the image characteristics of the targetobject have been intensified or otherwise enhanced. FIG. 8a illustratesthe result of this back-projection process applied to the exemplaryperspective view shown in FIG. 7.

FIG. 7 illustrates an exemplary perspective view displayed on a displaydevice, for example, showing an image captured by camera or videoequipment on a UAV. In this example, the target object in theperspective view is a red truck that has been selected by a user whoapplied a point 7 a on the truck using an appropriate input device. Inthis example, the system 100 identifies an area of interest 7 h based onthe user input and further determines a foreground area 7 c in theperspective view. The system 100 may generate histograms F(hist),B(hist), and NF(hist) as discussed above. FIG. 8a illustrates the areaof interest 7 b after each pixel value in the area of interest has beenreplaced with a corresponding value determined from the histogramNF(hist), for example, ranging in value from zero to one.

In some embodiments, the system 100 may further process theback-projected image to identify the contours of the target object. Forexample, the system 100 may use one or more filters to remove noise fromthe back-projected image created at step 450. Alternatively, or inaddition, the system 100 may normalize the back-projected image tomaximize its image intensity.

In an exemplary embodiment, the system 100 may optionally proceed tostep 460 to further identify the target object such as the contours ofthe target object. Using formula (1) and the assumptions discussedabove, the system may determine a threshold value where any pixels inthe back-projected image having a histogram value greater than thethreshold value likely contain information related to the target object.For example, in some embodiments, the system 100 may assume that F(hist)and B(hist) have the same weight for purposes of determining a thresholdvalue. In this example, the system may assume F(hist)=B(hist), soNF(hist)=0.5 based on formula (1), and therefore the system 100 may setthe threshold value equal to 0.5. Alternatively, the system 100 mayassign different weights to F(hist) and B(hist) to increase or decreasethe threshold value. For example, if the system assumesF(hist)=4×B(hist), then NF(hist)=0.8, then the determined thresholdvalue is equal to 0.8 in this different example.

If a pixel value in the histogram NF(hist) has a value greater than thethreshold value, then the system 100 may assume the pixel most likelycontains information related to the target object. Otherwise, for pixelvalues in histogram NF(hist) having a value less than or equal to thethreshold value, the system 100 assumes such pixels most likely containinformation related to the background area. Persons of ordinary skill inthe art will appreciate that, for purposes of these examples, theweights and threshold values have been arbitrarily defined for theconvenience of description.

In some embodiments, the system 100 may analyze the back-projected image(e.g., the back-projected image in FIG. 8a ) using the threshold value.At step 460, for example, the system 100 may reassign pixels in the areaof interest to define new foreground and background areas based on whichpixel values in the back-projected image exceed the threshold value. Inthis embodiment, the new foreground area may be defined by analyzing thepixel values in the area of interest (in the back-projected image) on apixel-by-pixel basis and comparing each of the pixel values with thethreshold value. If the pixel value is greater than the threshold value,then that pixel is reassigned to the new foreground area. As a result,the new foreground area may no longer correspond to the originalforeground area from step 420. Similarly, the new background area inthis disclosed embodiment may be defined on a pixel-by-pixel basis andmay not be coextensive with the originally defined background area fromstep 420. At step 460, the system 100 also may determine a new histogramNF(hist) based on the new F(hist) and B(hist) histograms, for example,using formula (1) above or another formula for combining the new F(hist)and B(hist) histograms.

In some embodiments where histogram NF(hist) was previously normalizedto a range between zero to one, the values stored in bins of the newhistogram NF(hist) also may be in the range of zero to one.Alternatively, the bins of the new histogram NF(hist) may comprise otherranges based on the image characteristics of the back-projected image.Regardless of the actual range of values, the system 100 may assign anypixels in the back-projected image (which was generated at step 450)having pixel values greater than the threshold value to the newhistogram F(hist). Otherwise, the system 100 may assign any pixels inthe back-projected image having pixel values equal to or less than thethreshold value to the new histogram B(hist). In this process, thesystem 100 is not bound by the previously defined foreground andbackground areas.

Furthermore, by analyzing the back-projected image in this way, thesystem 100 may minimize any errors introduced earlier in the process.For example, if the original foreground area was too small, then theoriginal foreground area did not capture the entire target object.Alternatively, if the original foreground area was too large, then theoriginal foreground area captured too much of the background features.By assigning pixels in the back-projected image to new foreground andbackground areas based on their probability of containing informationrelated to the target object, the system 100 may increase the chances ofidentifying the target object.

Having determined the new histogram F(hist) and new histogram B(hist),the system 100 may analyze the image characteristics or physicalmeanings of the first back-projected image (e.g., FIG. 8a ) to identifythe contours of the target object. In some embodiments, the system 100may use formula (1) again to determine the new histogram NF(hist). Inother embodiments, the system 100 may normalize the new histogramNF(hist). As discussed above, other suitable formulas may also oralternatively be used in embodiments consistent with this disclosure,and the new histogram NF(hist) need not be generated using the sameformula that was used to generate the original histogram NF(hist).

At step 470, the system 100 may back-project the new histogram NF(hist)onto the first back-projected image (e.g., FIG. 8a ) to obtain a secondback-projected image (e.g., FIG. 8b ). Alternatively, the system 100 mayback-project the new histogram NF(hist) onto the original area ofinterest (e.g., the area bounded by the perimeter 7 b in FIG. 7).Further, in some embodiments, the system 100 may use one or more filtersto remove noise, or alternatively the system 100 may normalize thesecond back-projected image to maximize its image intensity. In yetother disclosed embodiments, the system 100 may repeat steps 460 and 470through one or more additional iterations until the contours of thetarget object become more clearly identifiable in each generatedback-projected image.

At step 480, after the creation of an acceptable back-projected image,e.g. the n-th back-projected image where n is any number greater than orequal to one, the system 100 may create a binary representation of theacceptable back-projected image. In some embodiments, the system 100 mayuse a predetermined cutoff value to produce the binary image, such thatevery pixel value greater than the cutoff value may be assigned amaximum value and every pixel value less than or equal to the cutoffvalue may be assigned a minimum value. For example, FIG. 8c illustratesa binary representation of the exemplary back-projected image of FIG. 8b, where pixel values above a cutoff value were assigned a white colorand pixel values less than or equal to the threshold value were assigneda black color. Alternatively, the system 100 may apply a low-pass filterto the n-th back-projected image to determine the closest valley to themaximum value in the histogram, such that the position of such a valleyin the filtered image signal may be used to determine the cutoff valuefor generating the binary representation at step 480. In otherembodiments, the system 100 instead may determine an intensity histogramrepresenting the pixels of the n-th back-projected image in a range ofgrayscale.

The system 100 may extract the target object at step 490. In someembodiments, the system 100 performs a connected-component analysis onthe binary image created at step 480. For example, the system 100 mayassign certain identifiers to pixels in the binary image created at step480. Any pixel that is connected to another pixel (e.g., sharing aborder and having the same binary value) may be assigned the sameidentifier. Using this process, the system 100 may assign everyconnected component (e.g., region of adjacent pixels having the samebinary value) with a unique identifier. Other suitable methods ofperforming connected-component analysis may also or alternatively beused in embodiments consistent with this disclosure.

Once the system 100 has identified the connected components by assigningunique identifiers to different pixel regions, it may identify thetarget object. For example, the system 100 may identify which connectedcomponent contains the user-selected point (or is included in theuser-selected area) from step 410. The system 100 then may identify thetarget object as the object in the original perspective view that is ator near the same position of the area of interest as the identifiedconnected component. In some embodiments, the actual contour of thetarget object relative to the contour of the connected component may beused to identify the target object. In other embodiments, the system 100may define a new tracking perimeter around the target object, such asthe exemplary tracking perimeter around the red truck (target object) inFIG. 8d , based on which the system 100 can perform various trackingtasks.

Further to the disclosed embodiments above, the exemplary process 400allows the system 100 to accurately identify a target object based on auser selection. Moreover, the process 400 allows the system 100 tominimize the inclusion of background information during thetarget-tracking process. As a result, the process 400 provides a morerobust and accurate method of identifying and tracking target objects.

FIGS. 5 and 6 illustrate exemplary foreground and background areas thatmay be used for identifying a target object in accordance with thedisclosed embodiments. In some embodiments, a user may use a controller104 to select a target object on a display device. In some embodiments,the user may use an input device, such as a stylus, mouse, trackpad,etc., to input the user's selection to the user controller 104.Alternatively, the user may use a finger to touch a capacitivetouchscreen to enter the user's selection. In the exemplary embodimentshown in FIG. 5, the user selection represents the point 5 a on thetarget object, which is shown as a car for simplicity. The point 5 a maybe a single pixel or may be any other pixel width or shape depending onthe input device.

In some embodiments, the user input is communicated to a UAV 102 to beprocessed by hardware and/or software in the UAV. For example, the UAV102 may receive the user input and may use the user selection to processimages (including video frames) captured from the imaging device 106 toidentify a user-selected target object in those images, consistent withthe disclosed embodiments. Alternatively, the user input may beprocessed by hardware and/or software, such as an application, executingon the user controller 104, or alternatively processed by hardwareand/or software running on a mobile device (such as a smartphone,tablet, laptop, etc.) connected to the user controller 104, to identifythe target object consistent with the disclosed embodiments describedherein.

According to an exemplary embodiment, the system 100 may define aperimeter 5 b around an area of interest based on the user-selectedpoint 5 a. For example, the perimeter 5 b of an area of interest may becreated so the point 5 a is at or near the center of the area.Alternatively, the perimeter 5 b of an area of interest may bedetermined so the point 5 a merely has to be located anywhere within thearea. Using area within perimeter 5 b as the potential area of interest,the system 100 may determine another perimeter 5 c to define a targetarea intended to include the target object. In some embodiments, theperimeters 5 b and Sc defining areas are concentric. In otherembodiments, the perimeter 5 c is chosen to be at or near the center ofthe area 5 b. In other embodiments, the system 100 may select theperimeter 5 c at an arbitrary location within the perimeter 5 b.

According to other exemplary embodiments, the user may use the usercontroller 104 to select an area around a target object on a displaydevice. For example, the user may drag a cursor on the display device toselect (as shown by dashed arrow 6 a in FIG. 6) an area around a targetobject. In an exemplary embodiment, the system 100 may define the areaof interest based on the user-selected area, where the perimeter 5 b ofthe area of interest is the same as the boundary of the user-selectedarea. After defining the area of interest, the system 100 may create aperimeter Sc to define a foreground area as the target area. Theperimeter 6 c defining the target area may be at an arbitrary locationwithin perimeter 5 b of the area of interest.

As noted, FIGS. 7 and 8 show exemplary results of performing theexemplary technique for identifying a target object in accordance withthe disclosed embodiments. FIG. 7 shows an image, for example obtainedfrom a UAV, comprising a target object (e.g., the red truck) and variousother background objects (e.g., the gray and red taxi, greenlandscaping, blue sign, buildings, structures, etc.). FIG. 7 also showsan example of the user-selected point 7 a on the target object, andperimeters 7 b and 7 c. As shown in FIG. 7, the foreground area (e.g.,the area inside perimeter 7 c) contains the target object and variousbackground objects. On the other hand, the background area (e.g., thearea between perimeter 7 b and 7 c) contains only background objects,which includes a car that has a similar red color as the red truck whichis the target object. Perimeters 7 b and 7 c in FIG. 7 were definedconsistent with the disclosed embodiments.

FIG. 8a shows the resulting image that is created by applying steps 430to 450 in FIG. 4 to the exemplary perspective view shown in FIG. 7. FIG.8b shows the image that is generated by applying steps 460 to 470 to theback-projected image in FIG. 8a . FIG. 8c shows a binary image that mayresult by applying step 480 to the n-th back-projected image shown inFIG. 8b . FIG. 8d shows the final selection of the target object by thesystem 100 in this sequence of examples.

Other embodiments will be apparent to those skilled in the art fromconsideration of the specification and practice of the disclosedembodiments. It is intended that the specification and examples beconsidered as exemplary only, with a true scope and spirit of thedisclosed embodiments being indicated by the following claims. It is tobe understood that the examples and descriptions in this disclosure havebeen arbitrarily defined herein for the convenience of the description.The disclosed systems and methods are not limited to these simplifiedexamples, and other features and characteristics may be considered solong as the specified functions are appropriately performed.

While certain disclosed embodiments have been discussed with respect toUAVs for purposes of discussion, one skilled in the art will appreciatethe useful applications of disclosed methods and systems for identifyingtarget objects. Furthermore, although aspects of the disclosedembodiments are described as being associated with data stored in memoryand other tangible computer-readable storage mediums, one skilled in theart will appreciate that these aspects can be stored on and executedfrom many types of tangible computer-readable media. Further, certainprocesses and steps of the disclosed embodiments are described in aparticular order, one skilled in the art will appreciate that practiceof the disclosed embodiments are not so limited and could beaccomplished in many ways. Accordingly, the disclosed embodiments arenot limited to the above-described examples, but instead are defined bythe appended claims in light of their full scope of equivalents.

1.-34. (canceled)
 35. A controller configured to identify a targetobject in an image, the controller comprising: a memory storingexecutable instructions; and a communication interface configured toreceive the image; a user input device configured to receive a useridentification of a position of the target object in the image; and atleast one processor configured to execute the stored instructions to:define a first area surrounding the user identified position based onthe user identification and a second area surrounding the first area,wherein the first and second areas are concentric; and compare imagecharacteristics of the first area and the second area to identify thetarget object in the image.
 36. The controller of claim 35, wherein thecontroller is coupled to a display device configured to display theimage to a user.
 37. The controller of claim 36, wherein the displaydevice is integrated into the controller.
 38. The controller of claim36, wherein the display device is a touchscreen device.
 39. Thecontroller of claim 36, wherein the display device is an externaldisplay device that is connected to the controller.
 40. The controllerof claim 35, wherein the controller is implemented in a smartphone. 41.The controller of claim 35, wherein the communication interface isconfigured to receive the image from an unmanned aerial vehicle.
 42. Thecontroller of claim 35, wherein the image characteristics include atleast one of color, intensity, chrominance, luminance, brightness,lightness, or darkness.
 43. The controller of claim 35, wherein the useridentified position is a user-selected point in the image.
 44. Thecontroller of claim 35, wherein the user identified position is auser-selected area in the image.
 45. The controller of claim 35, whereinthe at least one processor is further configured to execute the storedinstructions to: generate a first histogram representing the imagecharacteristics in the first area and a second histogram representingthe image characteristics in the second area; determine a thirdhistogram by combining the first and second histograms using apredetermined function; and apply the third histogram to an area ofinterest, the area of interest comprising the first and second areas.46. The controller of claim 45, wherein the predetermined functiondetermines, for each image characteristic, a ratio of a value for thatimage characteristic in the first area divided by a value for that imagecharacteristic in both the first and second areas.
 47. The controller ofclaim 45, wherein the at least one processor is further configured toexecute the stored instructions to: redefine the first and second areasbased on probabilities that a plurality of points in the area ofinterest contain the target object; generate a new first histogramrepresenting the image characteristics in the redefined first area and anew second histogram representing the image characteristics in theredefined second area; and determine a new third histogram by combiningthe new first and second histograms using a second predeterminedfunction.
 48. The controller of claim 47, wherein the same predeterminedfunction is used to determine the third histogram and the new thirdhistogram.
 49. The controller of claim 47, wherein the at least oneprocessor is further configured to execute the stored instructions to:apply the new third histogram to the area of interest to create aback-projected image.
 50. The controller of claim 49, wherein the atleast one processor is further configured to execute the storedinstructions to repeat each of the redefining the first and secondareas, generating a new first histogram and new second histogram,determining a new third histogram, and applying the new third histogramto the area of interest.
 51. The controller of claim 35, wherein thefirst area is a foreground area and the second area is a background areasurrounding the foreground area.
 52. The controller of claim 35, whereinthe at least one processor is further configured to execute the storedinstructions to identify likely contours of the target object.
 53. Thecontroller of claim 35, wherein the at least one processor is furtherconfigured to execute the stored instructions to use the identifiedtarget object to track the target object's position.
 54. The controllerof claim 35, wherein comparing image characteristics comprises:determining a representation of a first physical meaning based on theimage characteristics in the first area; determining a representation ofa second physical meaning based on the image characteristics in thesecond area; and comparing the representations of the first and secondphysical meanings to identify the target object in the image. 55.-59.(canceled)